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1.  Introduction 


The  purpose  of  this  report  is  to  consolidate  lessons  learned  collected  from  13  SDCE 
applications  between  1993  and  1997,  and  synthesize  a  small  set  of  guidelines  for  SDCE  use. 
The  SDCE  methodology  is  described  in  full  detail  in  AFMC  Pamphlet  63-103  “Software 
Development  Capability  Evaluation”  [ref  1],  however,  the  description  of  SDCE  in  the 
AFMC  pamphlet  allows  many  choices  in  certain  areas.  The  choices  in  the  original  SDCE 
documentation  were  there  partly  by  design,  to  support  tailorability  of  the  method.  As  a 
result,  we  observed  variances  in  the  way  SDCE  was  applied  from  one  program  to  the  other, 
we  monitored  these  variances,  we  tracked  the  SPO’s  satisfaction  with  respect  to  the  choices 
made  for  the  variances  after  their  SDCE  application  was  completed,  and  we  used  this 
feedback  to  derive  the  guidelines  contained  in  this  document. 

The  guidelines  below  were  arrived  at  through  a  systematic  collection  of  metrics  and 
lessons  learned,  and  quantitative  and  qualitative  analyses  of  the  collected  data.  This  analysis 
was  performed  in  the  context  of  the  SDCE  Metrics  Mission  Oriented  Investigation  and 
Experimentation  (MOIE)  project.  In  general,  the  metrics  indicate  that  SDCE  has  been  an 
effective  tool  for  discriminating  among  bidders  and  predicting  software  risk  in  a  source 
selection.  Furthermore,  most  SDCE  applications  tracked  received  high  ratings  for  user 
satisfaction.  The  guidelines  contained  in  this  document  are  aimed  at  eliminating  occurrences 
that  resulted  in  low  user  satisfaction,  and  at  optimizing  future  SDCE  applications. 

One  of  the  features  that  distinguishes  SDCE  from  other  software  evaluation  methods  is 
the  fact  that  the  evaluation  is  integrated  with  the  overall  acquisition  process.  Therefore  a  key 
requirement  to  a  successful  SDCE  application  is  to  understand  the  system  acquisition 
framework  in  which  SDCE  is  performed,  and  the  relationships  between  the  SDCE  activities 
and  the  system  acquisition  activities.  Chapter  2  provides  an  overview  of  the  SDCE,  as  a 
reference.  Chapter  3  describes  the  system  acquisition  process  and  the  SDCE  process,  and 
relates  the  two. 

Another  feature  that  distinguishes  SDCE  from  other  software  evaluation  methods  is  the 
fact  that  it  is  highly  tailorable  to  the  needs  of  each  acquisition.  Chapter  4  contains  a 
discussion  of  the  variances  currently  allowed  in  the  SDCE  pamphlet,  when  the  choices  for 
each  of  the  variances  needs  to  be  made,  and  what  considerations  should  accompany  each  of 
these  choices,  and  reduces  the  SDCE  tailoring  process  to  a  sequence  of  six  steps. 

Chapter  5  contains  a  compilation  of  the  lessons  learned  from  the  13  SDCE  applications 
that  were  tracked,  and  organizes  them  with  respect  to  the  main  activities  that  constitute  the 
SDCE  process  as  described  in  chapter  2.  Chapter  6  identifies  the  questions  and  criteria  that 
were  most  frequently  asked  on  past  SDCEs.  These  are  offered  as  a  baseline  for  identifying  a 
core  subset  of  SDCE,  and  can  be  used  as  a  starting  point  for  SDCE  model  tailorings. 

Chapter  7  discusses  the  issues  associated  with  performing  SDCE  outside  the  source 
selection  framework.  Several  attempts  were  made  to  that  effect,  with  limited  success.  This 
chapter  explains  the  motivations  behind  these  attempts,  describes  the  difficulties 
encountered,  and  provides  recommendations  for  extending  SDCE  outside  the  source 
selection  framework. 

Chapter  8  contains  a  list  of  recommended  modifications  and  improvements  to  the  current 
version  of  AFMC’s  SDCE  pamphlet.  These  recommendations  are  based  on  the  guidelines 
described  in  previous  chapters.  As  a  result,  chapter  8  is  somewhat  redundant  with  the 
previous  chapters,  however,  its  purpose  is  to  consolidate  those  lessons  learned  that  amount  to 
modifications  to  the  current  version  of  the  AFMC  pamphlet. 
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Appendix  A  contains  sample  RFP  instructions  for  SDCE.  Appendix  B  contains  modified 
versions  of  the  SDCE  Evaluation  Templates  Appendices  A  and  B  reflect  the  lessons  learned 
and  guidelines  provided  in  chapters  4  and  5.  Appendix  C  lists  the  past  SDCEs  from  which 
the  guidelines  in  this  document  were  derived,  and  appendix  D  lists  the  acronyms  used. 

Throughout  the  report,  the  terms  “contractor”,  “offeror”  and  “bidder”  will  be  used 
interchangeably,  similarly  for  the  terms  “results”  and  “findings”. 
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2.  SDCE  Background 


SDCE  is  a  methodology  for  assessing,  during  source  selection,  contractors’  capabilities 
in  software  and  software-related  systems  engineering  disciplines.  The  SDCE  methodology  is 
based  on  the  assumption  that  contractors  who  have  identified  the  software-related  processes, 
plans,  tools  and  technologies  they  plan  to  use  on  an  upcoming  acquisition,  and  who  have^ 
prior  experience  in  using  them  present  a  lower  risk  to  the  acquisition  than  those  who  don  t. 
The  methodology  is  used  primarily  for  the  acquisition  of  software-intensive  systems  at  the 
Air  Force  Materiel  Command  (AFMC)  and  is  applied  in  the  context  of  a  source  selection. 

The  SDCE  assessment  consists  of  a  process  whereby  the  contractors  under  evaluation 
provide  information  about  their  capabilities  in  the  form  of  responses  to  SDCE  questions 
along  with  support  material  that  demonstrates  their  commitment  to  the  proposed  processes 
and  technologies,  and  their  past  experience  with  them.  The  contractors’  responses  are  then 
evaluated  using  predefined  evaluation  criteria,  and  validated  by  reviewing  the  support 
material  and  performing  a  site-visit. 

The  SDCE  questions  and  associated  criteria  are  organized  into  a  structure  that  is  referred 
to  as  the  SDCE  model.  The  SDCE  model  was  derived  from  Aeronautical  Systems  Center 
(ASC)’s  Software  Development  Capability  Capacity  Review  (SDCCR)  [ref  2]  and  the 
Software  Engineering  Institute  (SEI)’s  Capability  Maturity  Model  (CMM)  [ref  3].  It  covers 
a  broad  range  of  disciplines  and  is  meant  to  be  tailored  for  each  assessment,  based  on  the 
characteristics  and  requirements  of  the  program  for  which  the  assessment  is  performed. 

Every  application  of  SDCE  on  a  given  program  requires  a  tailoring  of  the  SDCE  model  to  the 
minimal  subset  of  the  model  that  will  adequately  represent  the  anticipated  risks  of  that 
program. 

The  SDCE  evaluation  is  performed  along  three  dimensions,  corresponding  to  three 
categories  of  inputs  from  the  offerors.  They  are: 

1)  Responses  to  SDCE  questions  describing  the  processes,  plans,  tools,  methods  and 
technologies  proposed  for  the  program.  These  are  evaluated  against  the  SDCE  criteria  and 
the  technical  requirements  of  the  acquisition  to  ensure  their  adequacy  with  respect  to  the 
acquisition’s  technical  requirements  and  programmatics. 

2)  Documentation  of  proposed  processes,  tools  and  plans  in  document(s)  that  are 
intended  to  be  used  in  the  acquisition  at  hand,  such  as  Software  Development  Plans,  Coding 
Standards,  etc. 

3)  Samples  from  past  projects  demonstrating  the  offerors’  experience  with  the  proposed 
processes,  tools  and  technologies. 

Note  that  categories  2)  and  3)  constitute  the  support  material  that  accompany  the  SDCE 
responses  with  the  proposal.  Support  material  is  reviewed  to  establish  a  measure  of 
confidence  in  the  offerors’  SDCE  responses.  The  support  material  is  usually  not  page 
limited. 

A  given  SDCE  application  entails  performing  the  activities  listed  in  figure  I.  This  list  is 
provided  as  a  reference  for  further  discussions  of  SDCE  experiences  and  lessons  learned. 

The  sequencing  of  SDCE  activities  is  illustrated  in  Figure  2.  More  details  about  SDCE  can 
be  found  in  the  SDCE  AFMC  Pamphlet  63-103  “Software  Development  Capability 
Evaluation”  [ref  1]. 
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♦  Determination  of  Applicability 

♦  Preparation  Phase 

E  Determine  Risks 

M  Develop  Eval.  Stds 
E  Tailor  Model 
¥  Prepare  RFP 
M  Prepare  Team 

♦  Evaluation  Phase 

P  Initial  Evaluation 

Site  Visit/CRs  DRs 
■  Final  Evaluation 

♦  Post-Award  Phase 
^Transition  Results 
S.  Conduct  Feedback 

li  Program  Follow  Through 


Figure  1.  SDCE  Activities 
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Flow 


Figure  2.  SDCE  Process  Flow 
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3.  Applying  SDCE  from  a  System  Acquisition  Perspective 


3.1  System  Acquisition  Process 

One  of  SDCE’s  unique  features  is  its  tight  coupling  to  the  specific  acquisition  for  which 
it  is  performed.  As  a  result,  the  SDCE  process  is  highly  dependent  on  the  overall  systern 
acquisition  process.  The  discussion  below  gives  a  few  details  about  the  system  acquisition 
phases,  SDCE  activities,  and  how  the  two  are  integrated.  A  more  complete  description  of  the 
system  acquisition  process  can  be  found  in  the  Federal  Acquisition  Regulations  (FAR)  [ref 
4],  and  the  Air  Force  FAR  Supplement  (AFFARS)  [ref  5]. 

The  system  acquisition  process  can  be  divided  into  four  main  phases:  the  acquisition 
strategy  planning  phase,  the  RFP  preparation  phase,  the  source  selection  phase,  and  the  post 
award  phase  (see  figure  3).  The  acquisition  strategy  planning  phase  corresponds  to  the 
identification  of  objectives  for  the  acquisition,  the  identification  of  the  main  risks  associated 
with  the  acquisition  and  the  development  of  a  strategy  for  the  acquisition.  Typical  outputs 
from  the  acquisition  planning  phase  are  a  Mission  Needs  Statement  (MNS),  User 
Requirements  (ORD)',  a  Single  Acquisition  Management  Plan  (SAMP),  a  Statement  of 
Objectives  (SOO),  an  Acquisition  Strategy  Panel  (ASP)  briefing,  etc. 

The  RFP  preparation  phase  corresponds  to  the  development  of  a  technical  requirements 
document,  and  the  development  of  evaluation  factors  and  criteria  for  selecting  a  “source”  for 
the  contract.  The  development  of  evaluation  factors  and  criteria  is  a  long  and  iterative 
process  that  consists  of  refining  and  prioritizing  the  evaluation  criteria,  and  organizing  them 
into  a  “source  selection  structure”  (which  corresponds  to  “Areas”,  “Factors”,  “Subfactors”, 
etc.).  The  RFP  preparation  phase  is  also  when  the  government  decides  what  material  needs 
to  be  received  from  each  offeror  with  respect  to  each  element  in  the  source  selection 
structure,  and  identifies  the  kinds  of  information  that  each  offeror  should  include  in  their 
proposals.  The  main  output  of  the  RFP  preparation  phase  is  the  RFP. 

The  source  selection  phase  is  when  the  government  evaluates  the  offerors’  proposals, 
selects  a  source,  and  establishes  one  or  several  contracts.  The  results  of  the  proposal 
evaluation  are  documented  in  the  form  of  strengths,  weaknesses,  and  risks  at  the  lowest 
levels  of  the  evaluation,  and  rolled  up  into  three  sets  of  ratings:  adequacy  ratings  (also  known 
as  “proposal  ratings”),  which  characterize  the  adequacy  of  the  proposal,  and  two  sets  of  risk 
ratings:  proposal  risk,  which  characterizes  the  risk  associated  with  the  proposal,  and 
performance  risk,  which  characterizes  the  risk  associated  with  the  offeror’s  past  performance. 
The  source  selection  phase  itself  can  comprise  one  or  three  subphases,  depending  on  whether 
the  government  decides  to  engage  in  discussions  with  the  offerors,  after  an  initial  evaluation 
of  the  proposals.  If  discussions  are  allowed  (“opened”),  then  a  discussion  subphase  takes 
place  for  clarifications  to  be  made  to  the  proposal.  The  discussions  subphase  is  then 
followed  by  a  final  evaluation  subphase.  If  discussions  occur,  they  are  formalized  through 
reports  called  Clarification  Requests  (CRs)  and  Deficiency  Reports  (DRs),  or  Engineering 
Notices  (EN).  The  collection  of  outputs  generated  during  the  evaluation  phase  is  referred  to 
as  the  source  selection  results,  which  are  archived  at  the  end  of  the  source  selection. 

Finally,  the  post  award  phase  corresponds  to  all  activities  that  take  place  after  a  source 
has  been  selected,  and  a  contract  is  in  place.  Most  of  the  activities  performed  by  the 
government  after  contract  award  are  aimed  at  maximizing  the  quality  of  the  delivered 
products  and  reducing  the  risk  of  cost  and  schedule  overruns. 


'  Although  initial  versions  of  the  MNS  and  the  ORD  are  usually  developed  prior  to  the  start  of  the  acquisition  to 
justify  its  undertaking,  some  updated  version  of  an  MNS  and  an  ORD  are  usually  associated  with  the  outputs  of 
the  Acquisition  Strategy  Planning  phase. 
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Although  these  four  phases  are  not  equal  in  terms  of  calendar  time  allocated  to  each  of 
them,  the  fourth  one  being  by  far  the  longest  one,  they  emphasize  the  early  part  of  an 
acquisition.  This  emphasis  is  consistent  with  the  belief  that  it  is  easier  to  eliminate  problems 
earlier  in  an  acquisition  than  later.  The  early  portion  of  an  acquisition  is  also  where  SDCE 
activities  are  concentrated. 


Figure  3.  System  Acquisition  Process 
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3.2  Relation  of  System  Acquisition  Process  to  SDCE  Process 


The  four  phases  of  SDCE,  (1)  Determination  of  SDCE  applicability,  (2)  SDCE 
preparation,  (3)  SDCE  evaluation,  and  (4)  Post  award  activities,  map  directly  to  the  four 
phases  of  the  system  acquisition  process  described  above  (see  figure  4).  In  particular, 
determination  of  SDCE  applicability  depends  on  whether  software  is  considered  to  be  a  risk 
for  the  acquisition,  a  determination  that  should  be  made  during  the  Acquisition  Strategy 
Planning  phase.  The  Preparation  phase  includes  a  determination  of  SDCE’s  position  in  the 
source  selection  structure,  the  development  of  SDCE  evaluation  standards,  and  other 
activities  whose  outputs  go  into  the  RFP;  therefore  it  should  be  performed  in  concert  with 
other  RFP  preparation  activities.  Similarly,  the  evaluation  phase  of  SDCE  produces  results 
that  feed  straight  into  the  source  selection,  its  activities  are  performed  in  parallel  with  other 
source  selection  activities,  and  three  out  of  the  four  activities  in  that  phase  (CR/DR  review. 
Site  Visits,  and  Final  Evaluation)  are  performed  only  if  the  source  selection  authority  decides 
to  engage  in  discussions  with  the  offerors.  Finally,  the  post  award  activities  of  an  SDCE 
correspond  to  a  desire  by  the  government  to  (1)  share  their  risk  findings  with  the  offerors 
(especially  the  successful  ones)  and  (2)  follow  up  on  these  risks  after  the  contract  has 
commenced,  a  desire  that  applies  to  software  risks  and  extends  to  all  risks  associated  with  the 
acquisition  at  hand. 


Figure  4.  SDCE  Within  the  System  Acquisition  Process 
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3.3  Consistency  of  SDCE  with  Acquisition  Reform 


We  conclude  this  chapter  with  a  few  remarks  about  acquisition  reform,  and  how  SDCE 
fits  into  the  reformed  acquisition  framework. 

The  main  focus  of  acquisition  reform  has  been  to  reduce  the  cost  of  system  acquisitions 
by  reducing  or  eliminating  those  government  activities  that  can  be  performed  by  the 
contractors,  and  by  increasing  trust  in  the  contractors  (Total  System  Performance 
Responsbility,  reduced  contractor  oversight,  etc.). 

SDCE  can  be  viewed  as  a  mitigator  for  the  risk  associated  with  increased  trust  in 
contractors.  As  the  government  moves  away  from  defining  detailed  technical  requirements, 
and  dictating  development  processes  to  the  contractors,  and  as  it  prepares  to  select  a 
contractor  that  it  will  trust  to  develop  a  system  with  little  oversight,  it  needs  increased 
visibility  into  the  plans,  processes,  and  technologies  identified  by  the  contractors  for  the 
project  at  hand  and  their  past  record  at  using  these  processes  and  technologies.  SDCE 
provides  a  structured  mechanism  for  contractors  to  describe  the  plans,  processes,  and 
technologies  identified  for  the  acquisition  at  hand,  along  with  their  past  record  in  them,  and 
for  the  government  to  review  them  and  establish  their  adequacy. 

In  addition,  SDCE  facilitates  the  early  identification  of  program  risks,  and  promotes 
government-contractor  dialogs.  SDCE  emphasizes  risk  management  in  that  the  purpose  of 
an  SDCE  is  to  identify  and  manage  software  risks  early  in  the  acquisition  process; 
furthermore,  the  SDCE  tailoring  is  driven  by  the  risks  associated  with  the  acquisition  at  hand. 
SDCE  encourages  open  dialog  with  the  contractors,  and  includes  activities  to  that  effect  at 
three  points:  during  the  planning  phases,  when  SDCE  is  briefed  to  the  prospective  contractors 
as  part  of  the  bidders’  conference;  during  the  evaluation  phase,  the  site  visit  is  meant  to  not 
only  obtain  detailed  information  about  the  contractors’  plans  and  processes,  but  also  foster 
cooperation  between  the  government  and  the  contractors;  finally,  at  the  conclusion  of  the 
evaluation,  when  debriefs  are  held  to  communicate  SDCE  results  to  each  of  the  offerors  and 
solicit  their  feedback. 
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4.  SDCE  Tailcring  Decisions 

The  SDCE  process  can  be  tailored  to  accommodate  the  needs  of  each  acquisition,  in 
terms  of  resources  and  schedule.  There  are  a  certain  number  of  points  during  that  process 
where  the  decisions  made  have  a  great  effect  on  the  impact  of  SDCE  and  the  resources 
required  to  perform  it.  These  decision  points  amount  to  the  tailoring  process.  They  are  listed 
in  this  chapter  along  with  the  dependencies  between  them. 

4.1  Primary  Tailoring  Decisions 

The  SDCE  tailoring  process  can  be  summarized  by  a  “how”  question  and  a  “what” 
question,  namely: 

•  How  is  SDCE  going  to  count  in  the  source  selection? 

•  What  software  risk  areas  are  most  applicable  to  the  acquisition,  and  therefore  should 
be  covered  by  SDCE? 

These  two  questions  can  map  into  six  variances  in  the  SDCE  process,  the  values  of  which 
need  to  be  planned  out  carefully.  These  tailoring  decisions  are: 

1.  the  position  of  SDCE  in  the  source  selection  structure, 

2.  the  total  number  of  questions  selected  for  the  evaluation, 

3.  the  selection  of  SDCE  questions, 

4.  the  selection  of  SDCE  team  members, 

5.  the  levels  at  which  SDCE  results  need  to  be  generated, 

6.  and  the  decision  to  perform  a  site  visit. 

The  following  paragraphs  explain  the  significance  of  the  primary  tailoring  decisions  and 
provide  recommendations  for  how  and  when  to  make  them. 

•  Position  of  SDCE  in  the  Source  Selection  Structure.  The  position  of  SDCE  in  the  source 
selection  structure  amounts  to  deciding  how  high  or  low  in  the  structure  SDCE  is 
positioned,  and  whether  to  keep  SDCE  as  one  entity  or  distribute  it  among  different 
entities.  SDCE  position  in  the  source  selection  structure  has  varied  greatly  in  the  13 
SDCEs  tracked.  Almost  every  possibility  was  tried  out;  this  provided  us  with  many  data 
points  and  lessons  learned  that  form  the  basis  of  the  following  recommendations.  The 
decision  of  where  in  the  source  selection  structure  SDCE  should  be  placed  should  be 
made  early  on  in  the  SDCE  preparation  phase,  because  the  scope  of  the  tailored  model 
should  be  consistent  with  the  importance  of  SDCE  in  the  source  selection,  which  in  turn 
is  determined  by  its  position  in  the  source  selection  structure.  For  example,  the  selection 
of  a  large  number  of  SDCE  questions  requires  a  large  team  to  evaluate  the  offerors’ 
responses  to  the  questions,  and  consumes  equally  large  resources  on  the  offerors’  side, 
therefore  it  should  not  correspond  to  an  SDCE  that  has  a  low  position  in  the  source 
selection  structure.  The  optimal  placement  of  SDCE  has  been  as  a  separate  factor  under 
the  technical  area. 

There  were  three  cases  among  the  tracked  SDCEs  where  SDCE  was  distributed  among 
different  areas  and/or  factors.  The  reason  why  SDCE  was  distributed  is  to  deconflict  it 
from  other  related  areas  and  factors.  In  all  cases,  this  resulted  in  great  difficulties  during 
the  evaluation  phase,  as  it  was  very  hard  to  coordinate  the  different  components  of 
SDCE.  Furthermore,  the  overall  impact  of  SDCE  on  the  source  selection  was  greatly 
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diminished  by  its  being  distributed.  Therefore  our  recommendation  is  that  SDCE  not  be 
distributed  among  different  components  of  the  source  selection.  Further  guidance  on 
how  to  deconflict  SDCE  from  other  portions  of  the  RFP  is  provided  in  section  5.2. 

•  Selection  of  SDCE  Questions.  The  selection  of  SDCE  questions  amounts  to  two  types  of 
decisions:  the  number  of  questions  selected  (i.e.,  the  size  of  the  SDCE  model),  and  the 
choice  of  questions.  The  total  number  of  SDCE  questions  selected  on  the  tracked  SDCE 
applications  ranged  from  27  to  1 18.  Our  recommendation  is  that  the  total  number  be 
proportional  to  the  importance  of  SDCE  in  the  source  selection,  and  its  place  in  the 
source  selection  structure.  The  main  reason  why  these  numbers  are  given  is  to  share  the 
fact  that  several  SDCE  applications  were  conducted  successfully  with  a  small  number  of 
questions  selected  (between  30  and  50  questions).  Observations  from  past  SDCE 
applications  indicate  that  the  more  people  are  involved  in  the  selection  of  SDCE 
questions,  the  larger  the  number  of  questions  selected;  this  is  probably  due  to  the  fact  that 
each  person  insists  on  having  their  “pet”  questions  included  in  the  selected  set.  The 
choice  of  questions  should  be  based  entirely  on  the  risks  associated  with  the  acquisition, 
and  not  on  personal  preferences.  Further  guidance  for  deciding  which  questions  to  select 
is  provided  in  chapter  5. 

•  Selection  of  SDCE  Team  Members.  SDCE  team  members  can  be  thought  of  as  being  in 
two  groups:  the  group  of  people  who  perform  the  planning  and  tailoring,  and  the  group  of 
people  who  perform  the  actual  evaluations.  The  reason  for  splitting  the  team  into  the  two 
subteams  is  that  the  selection  of  SDCE  evaluators  may  not  be  feasible  early  on,  because 
many  of  them  (if  not  all)  have  to  be  able  to  make  a  full-time  commitment  to  the  source 
selection  for  the  duration  of  the  initial  proposal  evaluation,  yet  the  schedule  of  the 
proposal  evaluation  is  not  determined  until  the  RFP  is  released,  and  the  date  of  the  RFP 
release  changes  a  lot  as  the  acquisition  progresses.  Another  difficulty  with  selecting  the 
SDCE  evaluators  in  advance  is  that  they  cannot  have  any  conflict  of  interest  with  any  of 
the  bidding  offerors,  yet,  in  many  cases,  the  offerors  are  not  known  with  certainty  until 
the  proposals  are  received.  So,  even  though  the  AFMC  pamphlet  shows  the  selection  of 
SDCE  team  members  as  one  of  the  early  activities  in  the  planning  phase,  experience  has 
shown  that  it  is  not  always  feasible  to  make  that  selection  too  much  in  advance  of 
proposal  receipt.  It  is  therefore  recommended  that  only  one  or  two  members  of  the  team 
be  selected  early  on  (a  planning  subteam)  to  work  on  the  tailoring,  RFP  preparation  and 
other  planning  activities.  Names  can  be  identified  for  the  rest  of  the  team  (the  evaluation 
subteam),  with  backups,  but  their  selection  cannot  be  finalized  until  the  schedule  for 
proposal  receipt  is  firmed  up,  which  is  usually  only  after  the  RFP  is  released.  Even  after 
the  RFP  is  released,  some  of  the  team  members  identified  for  the  evaluation  of  SDCE 
responses  may  end  up  disqualified  for  conflict  of  interest  reasons,  therefore  it  is  prudent 
to  have  a  few  extra  evaluators  lined  up  for  backup. 

•  Levels  at  Which  SDCE  Results  are  Developed.  Because  the  SDCE  model  is  structured 
into  5  layers,  with  the  questions  and  criteria  constituting  the  bottom  layer,  there  are 
several  options  for  the  level  at  which  the  atomic  results  should  be  generated,  and  how 
many  levels  they  should  be  rolled  up  to.  There  are  basically  two  options  for  the  level  at 
which  atomic  SDCE  results  are  generated:  the  question/criteria  level,  or  the  CC  level. 

The  question/criteria  level  is  hard  to  use  because  the  results  can  then  either  be  developed 
on  a  question-by-question  basis  or  on  a  criterion-by-criterion  basis.  However,  some 
questions  correspond  to  several  criteria,  and  some  criteria  map  to  several  questions, 
therefore  it  is  easier  to  document  the  results  for  an  entire  CC  at  once.  As  far  as  the  roll¬ 
up  is  concerned,  the  AFMC  pamphlet  provides  templates  for  documenting  SDCE  results 
at  each  level  of  the  SDCE  model,  i.e.,  at  the  Critical  Capability,  at  the  Critical  Capability 
Area,  at  the  Functional  Area,  and  finally  at  the  overall  SDCE  level  (no  templates  are 
provided  for  developing  results  at  the  question  and  criteria  level).  Experience  has  shown 
that  rolling  SDCE  results  from  one  level  of  the  model  to  the  next  offers  no  benefit;  in  fact 
it  tends  to  dilute  the  significance  of  the  atomic  results.  It  is  recommended  that  the  atomic 
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results  be  developed  at  the  CC  level,  and  that  the  only  roll-up  performed  be  at  the  overall 
SDCE  level.  This  model  was  successfully  used  on  the  past  two  SDCE  applications. 

•  Decision  to  Perform  a  Site  Visit.  The  SDCE  process  includes  a  step  where  site  visits  are 
performed  in  order  to  corroborate  the  responses  provided  by  the  offerors  and  to  obtain 
additional  clarifications.  Site  visits,  however,  are  optional;  they  can  be  conducted  only  if 
discussions  are  allowed  and  the  schedule  of  the  source  selection  permits  it.  Site  visits  are 
very  useful  because  they  facilitate  the  insight  needed  into  the  offeror’s  processes,  and 
they  promote  dialog  between  the  offerors  and  the  government.  Unfortunately,  they  are 
often  impractical  to  perform,  especially  when  there  are  numerous  offerors  who  are 
located  in  different  parts  of  the  country.  The  problem  is  exacerbated  by  the  fact  that  the 
site  visits  have  to  take  place  during  discussions,  and  discussions  are  often  held  during  a 
short  period  of  time  (two  weeks).  It  is  recommended,  however,  that  site  visits  not  be 
precluded.  The  bidders  should  be  prepared  to  host  site  visits,  and  the  final  decision  to 
hold  them  should  be  made  after  the  proposals  are  received  and  the  preliminary  evaluation 
is  completed. 


4.2  Dependencies  Between  the  SDCE  Tailoring  Decisions 

SDCE  activities  are  tied  to  other  source  selection  activities,  and  those  in  turn  are 
performed  in  a  highly  iterative  fashion,  therefore  it  may  not  be  possible  to  control  the 
sequencing  of  SDCE  activities  completely.  Care  should  be  taken,  however,  to  understand  the 
dependencies  between  the  different  tailoring  decisions  and  ensure  that  any  changes  made 
during  the  RFP  preparation  phase  to  any  of  them  preserves  the  consistency  of  the  overall 
SDCE  tailoring.  Again  the  main  example  is  when  the  position  of  SDCE  changes  by  reducing 
its  importance,  the  size  of  the  tailored  model  should  be  reduced.  We  have  often  observed  the 
reverse,  i.e.,  situations  where,  as  time  goes  by  during  the  RFP  planning  phases,  the  tailored 
SDCE  model  expands  while  the  position  of  SDCE  in  the  source  selection  structure  is 
reduced. 

The  position  of  SDCE  in  the  source  selection  structure  depends  on  the  importance  of 
software  to  the  acquisition  and  the  level  of  risk  associated  with  software.  The  size  of  the 
tailored  model  depends  on  the  position  of  SDCE  in  the  source  selection  structure,  and  the 
time  and  people  resources  available.  The  selection  of  questions  depends  on  the  technical 
requirements  associated  with  the  acquisition;  if  the  source  selection  is  a  downselection  and 
the  offerors  are  known,  there  may  be  additional  risks  associated  with  the  offerors  that  may 
drive  the  question  selection.  The  selection  of  team  members  must  be  consistent  with  the 
contents  of  the  tailored  model  in  order  to  ensure  that  expertise  relating  to  all  areas  selected  is 
represented  on  the  team;  it  also  depends  on  availability  of  team  members  and  the  conflicts  of 
interest  they  may  have  with  a  particular  offeror. 

The  levels  at  which  SDCE  results  are  generated  may  depend  on  the  position  of  SDCE  in 
the  source  selection  structure.  In  general,  it  is  recommended  that  SDCE  results  be  developed 
at  the  CC  level  and  rolled  up  only  at  the  overall  SDCE  level,  however,  that  may  not  always 
be  possible.  There  are  two  cases  where  this  rule  is  difficult  to  apply:  first,  if  SDCE  is 
distributed  among  several  areas  or  factors,  results  may  need  to  be  developed  at  different 
levels  corresponding  to  the  areas  and  factors  where  SDCE  is  distributed;  second,  if  SDCE  is 
an  Area  by  itself,  and  factors  are  identified  under  the  SDCE  area,  the  AFFARS  requires  that 
results  be  generated  at  the  factor  level,  and  the  factor  level  may  or  may  not  correspond  to 
CCs.  For  example,  if  there  are  too  many  CCs,  the  factors  may  correspond  to  groupings  of 
CCs,  and  results  will  need  to  be  generated  for  each  of  these  groupings.  Therefore,  some 
dependency  exists  between  the  decision  of  how  to  document  the  SDCE  results  and  the 
decisions  of  where  to  place  SDCE  in  the  source  selection  structure. 
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Finally,  the  decision  to  perform  a  site  visit  depends  only  on  the  schedule  of  the  source 
selection  and  the  duration  of  the  discussions;  it  is  independent  of  other  SDCE  planning 
decisions.  Site  visits  should  be  planned,  and  the  decision  to  perform  them  should  be 
postponed  until  the  last  possible  minute,  so  as  not  to  preclude  the  possibility  of  conducting 
them. 


Figure  5.  Dependencies  Between  the  6  SDCE  Tailoring  Decisions 
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5.  Recommendations 


The  following  are  lessons  learned  collected  from  various  people  who  participated  in  past 
SDCEs.  They  are  organized  by  SDCE  phase. 

5.1  Determination  of  Applicability  Phase 

5.1.1  Decision  to  Perform  an  SDCE 


If  software  is  not  a  major  risk  factor,  or  if  the  program  office  does  not  buy  in  to  the 
method,  SDCE  should  not  be  performed.  SPO  buy-in  is  not  always  easy  to  achieve.  We  had 
a  couple  of  situations  where  the  Procurement  Contracting  Officer  (PCO)  had  a  hard  time  with 
two  unique  features  of  SDCE:  (1)  the  large  volume  of  SDCE  support  material  sent  by  the 
offerors  and  (2)  the  site  visits.  Therefore,  it  is  important  to  explain  the  SDCE  process  to  the 
program  office  people  who  will  be  in  charge  of  the  source  selection,  including  the 
Procurement  Contracting  Officer  (PCO),  and  explain  the  SDCE-unique  features  and  their 
purpose.  If  there  is  insufficient  buy-in,  the  method  should  not  be  used. 

5.2  Preparation  Phase 


5.2.1  SDCE  Position  in  Source  Selection  Structure 


The  Source  Selection  Structure  is  a  hierarchical  structure  of  source  selection  criteria  used 
as  a  uniform  baseline  against  which  each  offeror’s  proposal  is  compared.  It  is  typically 
defined  in  terms  of  areas  and  factors,  or  factors  and  subfactors  (see  Figure  6).  By  its  nature, 
SDCE  always  relates  to  portions  of  the  proposal  (factors,  subfactors)  other  than  the  SDCE 
portion,  which  makes  it  difficult  to  place  SDCE  in  the  Source  Selection  structure.  In 
particular,  SDCE  almost  always  relates  to  the  Software  Technical  approach,  the  Software 
Management  approach,  and  the  System  Engineering  approach.  This  does  not  mean  SDCE 
needs  to  be  distributed  among  all  these  other  components.  Past  experience  has  shown  that 
evaluating  SDCE  as  an  integral  component,  i.e.,  a  separate  factor  or  subfactor,  facilitates  the 
evaluation  and  enhances  the  value  of  its  findings.  It  is  important  to  deconflict  SDCE  from 
related  portions  of  the  REP  without  damaging  the  integrity  of  the  evaluation.  This  can  be 
done  by  encouraging  communication  between  the  SDCE  team  and  teams  in  charge  of  other 
related  areas  and  factors.  In  general,  SDCE  should  be  placed  close  to  related  components  of 
the  source  selection.  By  order  of  relevance  to  SDCE,  software  technical  approach  is  the  most 
relevant  other  portion,  followed  by  systems  engineering. 
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Figure  6.  Example  of  SDCE  Position  in  Source  Selection  Structure — Scheme  1 


Two  schemes  are  presented  here.  One  scheme  is  based  on  past  experience  that  utilized 
the  old  version  of  the  AITARS.  A  solution  that  has  worked  in  the  past  is  to  have  the 
Software  Technical  Approach  and  SDCE  as  two  factors  under  the  Technical  Area  (see  figure 
6).  Another  scheme,  illustrated  in  figure  7,  is  to  place  SDCE  in  the  “Performance 
Confidence”  part  of  the  source  selection  structure,  if  such  a  structure  is  adopted.  The  first 
scheme  is  based  on  successful  past  experiences,  while  the  second  scheme  is  derived  from 
early  versions  of  the  AFFARS  rewrite. 
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Figure  7.  Example  of  SDCE  Position  in  Source  Selection  Structure — Scheme  2 
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5.2.2  Model  Tailoring 


Model  tailoring  refers  to  the  activity  of  taking  the  717-question  SDCE  questionnaire,  and 

reducing  it  to  a  small  set  of  questions  that  reflect  the  high  value  discriminators  for  the 

acquisition  at  hand. 

•  Process  of  selecting  the  SDCE  questions.  The  process  of  selecting  the  SDCE  questions, 
i.e.,  the  process  used  to  tailor  the  model,  offers  different  possibilities  regarding  the 
number  of  people  involved  in  the  tailoring,  the  use  of  tailoring  tools,  and  the  use  of 
“core”  questions.  The  more  people  are  involved  in  the  selection  of  SDCE  questions,  the 
less  efficient  that  process  becomes,  and  the  larger  the  tailored  model  ends  up  being.  It  is 
therefore  recommended  that  a  small  group  of  people  (one  or  two)  tailor  the  model,  and 
then  get  their  tailoring  reviewed  by  a  large  number  of  people  from  the  SPO  representing 
each  of  the  other  areas  of  the  RFP.  This  ensures  buy-in  from  the  SPO  and  helps  keep  the 
tailored  model  consistent  with  other  portions  of  the  RFP.  There  are  a  couple  of  aids 
available  during  the  tailoring  activities:  several  tailoring  tools  were  developed  at  SMC, 
ASC  and  Aerospace  [ref  6]. 

There  are  14  questions  that  have  been  identified  as  core  questions  (see  chapter  6).  It  is 
recommended  that  the  first  step  in  the  tailoring  be  the  selection  of  a  target  number  for  the 
total  size  of  the  tailored  SDCE  questionnaire.  As  a  second  step,  the  core  questions  can  be 
used  as  a  baseline.  Additional  questions  can  then  be  selected,  based  on  the  technical 
requirements  and  risks  associated  with  the  acquisition,  and  consistently  with  the  target  for 
total  number  of  SDCE  questions.  One  way  to  do  that  is  to  generate  a  list  of  software 
risks,  map  the  software  risks  to  the  FAs,  CCAs,  CCs,  then  select  the  actual  questions 
from  each  CC.  The  final  step  is  to  review  the  overall  set  of  questions  to  make  sure 
dependencies  between  questions  across  the  CCs  are  preserved. 

•  Modifications  to  the  SDCE  model.  When  the  SDCE  AFMC  pamphlet  was  first 
completed,  it  was  recognized  that  the  model  would  evolve  based  on  the  fact  that  it 
represents  the  most  common  software  risks,  which  in  turn  are  derived  from  lessons 
learned  and  new  technologies.  Both  the  lessons  learned  and  the  new  technologies  evolve 
with  time,  and  so  the  model  needs  to  evolve.  For  purposes  of  consistency  between 
different  applications  of  the  method,  and  to  keep  some  control  over  the  quality  of  SDCE 
applications,  the  AFMC  pamphlet  restricts  the  modifications  that  can  be  made  to  the 
model  to  additions  or  deletions  only.  Furthermore,  before  a  new  CCA  or  CC  is 
developed  for  an  acquisition,  the  AFMC  SDCE  OPR  should  be  contacted  to  determine  if 
CCAs  or  CCs  similar  to  those  needed  have  already  been  developed  by  some  other 
organization.  Consistently  with  these  recommendations.  Aerospace  developed  new 
sections  for  the  technology  area  FA6.  The  Aerospace  augmented  SDCE  model  was 
succesfully  used  on  several  SDCEs,  and  is  documented  in  [ref  7]. 

When  new  questions  and  criteria  are  added,  care  should  be  taken  to  focus  the  questions 
and  include  as  much  detail  as  possible  in  them.  The  questions  should  be  viewed  as  a  tool 
for  soliciting  offeror  inputs  that  come  in  the  form  of  responses  to  the  SDCE  questions. 
Questions  that  are  general  may  generate  responses  that  lack  the  focus  and  detail  needed  to 
complete  the  assessment.  It  should  not  be  assumed  that  details  or  clarifications  will  be 
obtained  later,  as  that  may  not  be  the  case:  sometimes  discussions  are  not  allowed.  Even 
when  discussions  are  allowed,  the  number  of  clarifications  requested  is  limited  to  critical 
areas  where  the  existing  offeror  material  indicates  the  possibility  of  serious  problems,  not 
areas  where  the  government  did  not  request  the  right  kind  of  material. 
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5.2.3  Evaluation  Standards 


Evaluation  standards  establish  the  level  an  offeror’s  proposal  must  meet  in  any  area, 
factor  or  subfactor  to  be  judged  acceptable  (“green”).  The  evaluation  standards  associated 
with  SDCE  should  be  written  in  such  a  way  as  to  capture  the  3  elements  of  an  SDCE, 
namely: 

•  Soundness  of  the  choices  of  software  processes,  methodologies,  tools  and  technologies 
for  the  program  at  hand; 

•  Completeness  of  the  proposed  plans  and  their  consistency  with  the  proposed  approach; 

•  The  existence  of  past  experience  in  the  proposed  methods,  tools  and  technologies, 
alternatively,  a  good  justification  for  proposing  new  approaches. 

Furthermore,  the  level  of  detail  in  the  evaluation  standards  associated  with  SDCE  should 
be  consistent  with  the  position  of  SDCE  in  the  source  selection  structure,  and  their  content 
should  be  consistent  with  the  tailored  model. 

5.2.4  Bidders’  Conference 


It  is  recommended  that  SDCE  be  briefed  to  the  contractors  as  part  of  the  bidders’ 
conference.  This  promotes  early  dialog  with  the  contractors  regarding  their  software 
proposal,  and  helps  the  contractors  understand  SDCE. 


5.2.5  Instructions  to  Offerors 


Instructions  to  offerors  must  be  detailed  yet  clear.  Four  specific  tips  are  included  below: 

•  Executive  Summary.  In  addition  to  responses  to  the  SDCE  questionnaire,  a  paragraph  or 
two  describing  the  organization  of  the  offeror  team,  the  allocation  of  software  functions 
among  primes  and  subs,  and  the  general  characteristics  of  the  software  (number  of  lines 
of  code,  language  used,  etc.)  can  be  very  helpful  to  the  evaluators,  and  should  not  be  hard 
to  generate  by  the  offerors.  This  reduces  the  need  for  evaluators  to  go  digging  in  to 
different  portions  of  the  proposals,  and  minimizes  the  risk  that  the  evaluators  will  miss 
some  important  information  about  the  offeror’s  software  approach. 


•  Use  of  page  limits.  Technically,  no  portion  of  the  SDCE  volume  should  be  page  limited, 
however,  that  is  not  always  practical  or  acceptable  to  the  program  office.  If  limits  need  to 
be  established,  the  following  guidelines  should  be  followed:  The  support  material  should 
not  be  page  limited,  because  it  often  comes  from  pre-existing  documents,  which  should 
not  be  modified  for  the  puipose  of  SDCE.  There  are  two  kinds  of  limits  that  can  be  used: 
(1)  the  number  of  past  project  samples  for  a  given  CC  can  be  limited,  we  recommend  1-2 
samples/CC);  and  (2)  the  total  number  of  pages  for  SDCE  responses  can  be  limited;  it  is 
recommended  that  the  total  number  of  pages  for  SDCE  responses  equal  one 
page/question,  and  that  the  offerors  be  given  the  freedom  to  allocate  the  total  number  of 
allowed  pages  among  the  different  questions  as  they  see  fit. 


•  Software  Development  Plan(s).  If  an  SDP  is  not  required  with  the  proposal,  it  should  at 
least  be  suggested  as  a  good  document  to  provide  among  the  SDCE  support  material. 
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•  Templates  for  SDCE  support  material.  Several  problems  were  experienced  with  the  RFP 
templates  provided  in  the  AFMC  pamphlet.  In  particular,  the  project  cover  sheet 
template  contained  insufficient  information  about  the  technical  relevance  of  the  project 
sample  provided.  The  support  material  was  not  adequately  cross-referenced  to  the  CCs, 
and  some  of  the  military  specifications  and  standards  mentioned  in  the  template  headers 
are  no  longer  applicable.  As  a  result,  the  templates  have  been  updated,  the  updated 
templates  have  been  successfully  used  on  more  than  five  source  selections;  they  are 
provided  in  chapter  7,  and  in  appendix  A.  It  is  recommended  that  the  updated  RFP 
templates  be  used  in  lieu  of  those  provided  in  the  SDCE  AFMC  pamphlet. 


The  reader  is  referred  to  appendix  A  for  an  example  of  SDCE  Instructions  to  Offerors 
that  incorporates  the  above  recommendations. 

5.2.6  Training 


The  skills  required  to  perform  SDCEs  fall  into  two  categories:  knowledge  of  the  SDCE 
method  and  process,  and  knowledge  of  the  technical  subject  matter.  SDCE  training  material 
is  available  in  a  variety  of  forms:  AFMC  offers  a  three-day  SDCE  seminar  that  can  be 
scheduled  through  the  SDCE  OPR  for  AFMC,  and  The  Aerospace  Institute  offers  a  three- 
hour  SDCE  course,  and  a  one-hour  SDCE  Overview.  Regarding  the  technical  skills,  the 
criteria  used  to  select  SDCE  evaluators  should  include  technical  expertise  in  the  software 
subject  matter,  a  skill  that  cannot  be  taught  in  a  short  course,  and  for  which  extensive 
experience  is  required.  The  only  additional  technical  training  needed  is  training  in  the 
technical  requirements  of  the  acquisition  at  hand.  It  is  highly  recommended  that  all  members 
of  a  given  SDCE  team  be  provided  with  the  technical  requirements  for  the  acquisition  at 
hand  prior  to  the  start  of  the  proposal  evaluation  (Operational  Requirements  Document,  or 
Technical  Requirements  Document,  or  A-Specifications,  etc.),  and  that  they  be  encouraged  to 
read  them. 


5.3  Evaluation  Phase 

5.3.1  Allocation  of  SDCE  Questions  to  SDCE  Evaluators 

Allocation  of  SDCE  questions  to  SDCE  evaluators  is  typically  done  based  on  the 
availability  of  evaluators  and  their  technical  strengths.  The  following  guidelines  should  be 
observed:  each  question  should  have  at  least  double  coverage  in  order  to  ensure  that  no  input 
from  the  bidders  was  overlooked;  not  all  evaluators  need  to  make  a  full-time  commitment  to 
the  SDCE  team,  however,  it  is  important  to  have  a  core  set  of  2  to  3  people  who  are 
dedicated  to  the  evaluation  for  its  duration;  finally,  our  observations  are  that  one  evaluator 
can  review  at  most  three  CCs  per  day  per  offeror,  and  an  allocation  of  two  CCs  per  day  per 
offeror  is  advised. 

5.3.2  Scheduling  of  Evaluation  Activities 

The  evaluation  activities  that  need  to  be  scheduled  are  the  start  and  completion  of  the 
individual  evaluations,  the  discussion  sessions,  and  the  writing  of  an  SDCE  report,  if  one  is 
written.  The  purpose  of  the  discussion  sessions  is  to  discuss  the  findings  of  the  individual 
evaluators  with  the  rest  of  the  group,  primarily  to  come  up  with  an  integrated  set  of  results, 
but  also  to  share  information,  especially  in  situations  where  inputs  for  one  CC  are  relevant  to 
another  one,  as  is  very  often  the  case.  It  is  recommended  that  a  general  discussion  session  be 
held  at  least  once  for  each  offeror.  The  duration  of  such  a  discussion  depends  on  the  size  of 
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the  tailored  model.  Based  on  past  SDCEs,  one-half  day  discussion  session  per  offeror  is 
sufficient  for  a  40-question  SDCE.  Developing  an  SDCE  report  is  not  required,  however,  it 
can  be  useful  to  capture  the  SDCE  findings  before  they  get  integrated  with  the  rest  of  the 
source  selection  findings,  and  generating  a  report  is  one  mechanism  for  doing  that.  The 
report  can  contain  a  documentation  of  the  strengths,  weaknesses,  and  risks  identified  for  each 
offeror,  and  any  other  context  information,  such  as  the  composition  of  the  offeror’s  team,  the 
quality  and  presentation  of  their  SDCE  input,  and  any  unique  features  contained  in  their 
SDCE  volume. 

5.3.3  Site  Visits 


If  site  visits  are  allowed,  they  should  be  focused  on  the  weaknesses  and  the  areas  needing 
clarifications,  and  prioritized  according  to  their  risk  levels.  Contractors  should  be 
discouraged  from  giving  open-ended  presentations  or  demonstrations;  instead,  a  detailed 
agenda  should  be  sent  by  the  government.  The  schedule  for  the  site  visit  should  allow  one 
day  for  the  site  visit,  half  a  day  for  analyzing  the  information  provided  during  the  site  visit, 
and  another  half-day  to  verify  the  additional  information  with  the  contractors,  for  a  total  of 
two  days  plus  travel  time. 

5.3.4  Evaluation  Results 

The  process  for  developing  source  selection  evaluation  results  is  mandated  by  the  FAR 
[ref  4]  and  the  AFFARS  [ref  5].  It  consists  of  a  build-up  that  starts  with  atomic  elements 
consisting  of  strengths,  weaknesses  and  risks,  which  are  then  rolled  up  to  the  subfactor  or 
factor  level  and  associated  with  symbolic  ratings  (e.g.,  proposal  ratings  of  blue,  green, 
yellow,  red;  or  risk  ratings  of  High,  Medium,  Low).  A  strong  emphasis  is  placed  in  the  FAR 
and  the  AFFARS  on  the  importance  of  generating  detailed  write-ups,  also  known  as  narrative 
assessments,  to  justify  every  component  in  the  roll-up  process  from  the  initial  strengths, 
weaknesses  and  risks  to  the  final  symbolic  ratings. 


Figure  8.  Suggested  Mapping  of  SDCE  Inputs  to  Proposal  Rating  Categories 
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There  are  two  problems  that  can  face  SDCE  evaluators  in  the  development  of  SDCE 
results:  how  to  decide  which  SDCE  results  constitute  Weaknesses  versus  Risks,  and  how  to 
map  the  templates  provided  in  the  SDCE  pamphlet  to  the  AFFARS  format.  Since  the 
Strengths  and  Weaknesses  components  of  the  evaluation  get  rolled  up  into  the  adequacy 
rating  (or  “proposal  rating”),  which  represents  the  adequacy  and  soundness  of  the  proposed 
approach,  and  the  risk  components  get  rolled  up  into  the  Risk  Rating  (also  known  as 
“proposal  risk”),  which  is  an  attribute  of  the  approach  corresponding  to  the  level  of  risk 
associated  with  the  proposed  approach,  it  is  suggested  that  the  evaluation  of  questionnaire 
responses  be  used  to  develop  the  adequacy  rating  (proposal  rating)  and  that  the  support 
material  be  used  to  develop  the  risk  rating  (proposal  risk).  This  paradigm  is  illustrated  in 
figure  8. 

The  AFMC  SDCE  pamphlet  contains  templates  for  developing  SDCE  results,  however, 
the  information  in  the  SDCE  templates  is  not  identical  to  the  information  required  by  the 
AFFARS.  In  particular,  the  templates  in  the  pamphlet  contain  a  field  for  adequacy  ratings 
with  values  of  Strong,  Moderate,  or  Weak  (S,  M,  W),  and  a  field  for  risk  ratings  with  values 
of  High,  Medium,  or  Low  (H,  M,  L).  The  SDCE  templates  also  ask  for  comments  to  go 
along  with  the  ratings.  The  AFFARS,  on  the  other  hand,  asks  for  narrative  assessments  to 
justify  the  strengths,  weaknesses,  and  risks.  Furthermore,  the  proposal  rating  required  in  the 
AFFARS  is  a  four- valued  rating.  The  two  formats  need  to  be  mapped  such  that  the 
comments  field  in  the  SDCE  template  is  used  to  document  the  narrative  assessments,  and  the 
proposal  rating  and  risk  assessment  in  the  templates  are  given  the  same  number  of  values  as 
defined  in  the  applicable  regulations  (Air  Force  regulations  use  four  valued  color  ratings  for 
proposal  ratings).  New  evaluation  templates  were  developed  for  our  more  recent  SDCE 
applications.  They  are  provided  in  Appendix  B  “SDCE  Evaluation  Templates”. 
AERO_SDCE  is  a  tool  that  was  developed  to  automate  the  management  and  roll-up  of  SDCE 
result  documentation  [ref  8]. 


5.4  Post- Award  Phase 

5.4. 1  Post  Award  Activities. 


This  is  by  far  the  most  neglected  of  all  SDCE  activities.  One  reason  why  the  SDCE- 
identified  risks  are  not  followed  up  is  that  the  SDCE  results  get  lost  after  they  have  been 
tightly  integrated  with  other  source  selection  results,  so  it  is  not  easy  to  recover  the  original 
SDCE  findings  after  the  source  selection  is  over.  This  is  especially  true  if  SDCE  does  not  get 
a  separate  set  of  ratings  in  the  source  selection,  such  as  when  it  is  distributed  among  different 
areas  and  factors,  or  when  it  is  too  low  in  the  source  selection  structure.  Another  reason  is 
that  the  source  selection  results,  including  SDCE  results,  are  source  selection  sensitive,  and 
therefore  become  hard  to  access  once  the  source  selection  is  over.  A  contributing  factor  is 
the  change  in  personnel,  which  often  happens  with  the  start  of  a  new  contract.  Three  things 
can  be  done  to  help  ensure  post-award  SDCE  follow-up: 

1.  The  SPOs  should  be  educated  about  the  importance  of  software  risk  in  general  and 
the  importance  of  following  up  on  the  SDCE-identified  risks  in  particular. 

2.  The  transition  of  SDCE  results  outside  the  source  selection  should  be  planned.  This 
is  facilitated  by  documenting  the  SDCE  results  in  a  separate  report  and  declassifying  the 
portion  of  the  report  that  adS'esses  the  successful  offeror.  Aero_SDCE  is  a  tool  that  manages 
the  documentation  and  integration  of  SDCE  results  with  source  selection  results  such  that  the 
original  SDCE  findings  are  preserved  [8]. 
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3.  The  RFP  should  include  provisions  for  post-award  SDCE  follow-up.  The  provisions 
can  include  a  post-award  follow-up  SDCE  (see  chapter  7,  SDCE  outside  the  Source 
Selection),  and  award  fee  criteria  for  SDCE. 
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6.  C(H*e  Questions 


This  section  provides  a  core  set  of  SDCE  questions  and  criteria.  The  proposed  set 
corresponds  to  the  questions  that  were  most  frequently  asked  on  9  past  SDCEs  for  which 
question  selection  was  tracked.  This  set  is  proposed  only  as  a  starting  point  when  selecting 
SDCE  questions  for  a  particular  tailoring,  additional  questions  should  be  selected  based  on 
the  program’s  risk  profile.  The  reader  is  referred  back  to  section  5.2.2  for  model  tailoring 
guidelines. 

6.1  List  of  Core  Questions  and  Criteria 

The  following  lists  14  questions  that  were  the  most  frequently  used  questions  on  the  past 
SDCEs  that  were  tracked. 

CC  1.3.2:  Subcontractor  Management 

*  Question  1: 

Fully  describe  your  process  for  subcontractor  management  including  reporting 
and  control  of  the  subcontractor  software  development  activities.  How  does  this 
process  relate  to  and  integrate  with  your  overall  system  program  management 
approach?  Describe  how  the  subcontractor  management  and  review  activities  are 
reflected  in  the  program  level  system  engineering  planning  documents 
(IMP/IMS/ITAMP).  Cl 

Criterion  1: 

The  proposed  subcontractor  management  process  is  integral  to  the  system 
program  management  process  and  provides  integrated  reporting  and  control  of  the 
subcontractor  software  development  activities  consistent  with  the  program’s 
management  control  system. 

CC  2.1.5:  Systems  Requirements  Management 

*  Question  1: 

Describe  the  process  used  to  provide  two-way  requirements  traceability.  At  what 
point  is  requirements  traceability  established  and  documented?  What  provisions  exist 
to  maintain  the  traceability?  Cl 

Criterion  1: 

Two-way  requirements  traceability  is  maintained  from  system  specifications  to 
hardware  and  software  configuration  item  specifications. 

CC  2.6.1:  Test  and  Integration 

*  Question  4: 

Describe  any  special  integration  and  test  plans  developed  for  commercial-off-the- 
shelf  (COTS)  software  or  other  reuse  software?  C4 

Criterion  4; 

Any  use  of  commercial -off-the-shelf  (COTS)  software  or  other  reuse  software  is 
incorporated  into  system  integration  and  test  planning. 
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2.7.1:  Reuse 

*  Question  2: 

What  trade-off  studies  have  been  done  or  are  planned  to  evaluate  the  costs, 
benefits,  and  risks  of  the  opportunities  to  reuse  existing  system  and  software 
components?  Cl 

Criterion  1: 

Opportunities  to  utilize  previously  developed  system  and  software  components 
(including  architectures,  designs,  code,  and  documentation)  are  identified  and  subject 
to  trade-off  studies. 

CC  3.3.1:  Software  Requirements 

*  Question  1: 

Describe  the  software  requirements  analysis  process  to  be  applied.  Identify  the 
specific  methodologies  and  tools  to  be  used  to  support  the  analysis  process.  What 
organizational  element  is  responsible  to  perform  the  analysis?  Identify  the  input  to 
and  output  product  from  the  analysis.  C3 

Criterion  3: 

The  selected  requirements  analysis  methodology  is  compatible  with  other 
methodologies  applied  on  the  program.  The  analysis  methodology  is  supported  with 
necessary  tools. 

CC  3.3.2:  Software  Requirements 

*  Question  1: 

Describe  the  software  development  activities  that  result  from  a  change  in  or 
addition  to  the  requirements.  When  do  they  get  performed?  How  do  you  ensure  that 
they  are  performed?  Cl 

Criterion  1: 

The  software  development  artifacts  (requirements,  design,  code,  documentation) 
are  revised  as  changes  to  the  requirements  are  incorporated. 

CC  3.4.1:  Software  Design 

*  Question  1: 

Describe  the  process  and  specific  methodologies  used  to  develop  the  top-level  and 
detailed  software  design.  Is  the  same  methodology  used  to  maintain  the  design 
through  development  and  life  cycle  support?  What  tools  are  used  to  support  the 
methodology?  Cl 

Criterion  1: 

A  methodology  is  used  to  develop,  document  and  maintain  the  top-level  and 
detailed  software  design. 
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CC  3.4.1:  Software  Design 

*  Question  3: 

What  mechanism  and  format  are  used  to  describe  the  execution  priorities  of  the 
different  components,  and  the  execution  control?  C2 

Criterion  2: 

The  design  description  includes  the  (static)  structure  and  the  (dynamic)  behavior 
of  the  software. 

CC  3.5.2:  Software  Coding 

*  Question  2: 

Describe  your  process  for  estimating  the  effect  of  code  changes  on  other  parts  of 
the  system,  including  variables  and  other  software  components.  What  tools  are  used? 
Who  is  involved  in  the  process?  C2 

Criterion  2: 

Code  changes  are  reviewed  for  correctness,  and  to  avoid  undesired  impact  on 
other  software  and  system  variables  and  components. 

CC  3.6.1:  Software  Test  and  Integration 

*  Question  1: 

Describe  your  process  for  planning  the  software  integration.  How  many  different 
components  do  you  integrate  at  once?  How  do  you  determine  the  order  for 
integrating  the  different  software  components?  Describe  how  your  integration 
process  accommodates  all  levels  of  software  integration.  Cl  C2  C4 

Criterion  1: 

The  software  integration  planning  takes  into  account  the  interdependencies 
between  the  different  software  components  and  the  criticality  of  each  component. 

Criterion  2: 

The  software  integration  planning  takes  into  account  the  availability  of  other 
components  of  the  system. 

Criterion  4: 

The  software  integration  planning  and  process  accommodate  software  integration 
starting  with  the  lowest  level  elements,  i.e.,  units  through  all  levels,  including  CSCI 
and  CSCI/HWCI. 


29 


CC  3.6.2;  Software  Testing 

*  Question  2: 

What  tools  will  be  used  for  testing?  When  will  they  be  available?  Will  they 
require  any  special  inputs?  Will  their  outputs  require  any  special  processing?  What  is 
your  process  to  ensure  that  all  required  test  resources  have  been  planned  and 
allocated?  C2 

Criterion  2: 

A  process  exists  to  ensure  that  software  testing  is  adequately  planned  with 
sufficient  test  resources. 

CC  4.4.2:  Metrics 

*  Question  1: 

Identify  the  metrics  you  plan  to  collect  on  this  program,  which  system  or 
software  product  they  apply  to,  which  process  they  apply  to  and  or  what  progress  they 
measure.  Cl 

Criterion  1: 

The  metrics  selected  for  the  program  address  the  system  and  software  products, 
the  process  used  to  generate  the  products,  and  the  progress  of  the  development  effort. 

CC 4.7.2:  Configuration  Management 

*  Question  5: 

What  is  the  program  approach  to  establishing  and  controlling  developmental 
baselines  and  test  configurations?  C4 

Criterion  4: 

Procedures  exist  and  are  followed  to  create  and  maintain  developmental  builds 
and  incremental  test  baselines. 


CC  5.7.2:  S/SEE 
*  Question  4: 

For  each  tool  in  the  S/SEE,  describe  its  functionality,  its  maturity,  the  quality  of 
its  documentation,  and  how  it  will  be  supported  during  the  program.  Explain  the 
rationale  for  selecting  new  (not  yet  matured)  tools  and  how  confidence  is  established 
in  the  ability  of  these  new  tools  to  meet  program  needs.  Cl  C2 

Criterion  1: 

The  S/SEE  components  support  the  program’s  software  engineering  development 
and  management  requirements,  functions,  methodologies,  and  activities. 

Criterion  2: 

The  S/SEE  components  are  mature  and  well  documented.  New  tools  are 
determined  through  systematic  evaluation  to  meet  program  needs. 
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6.2  Process  Used  to  Derive  Core  Questions 

SDCE  question  selection  was  tracked  on  9  programs.  For  each  question  in  the  717 
question  model,  a  number  was  computed,  the  “number  of  selections”.  This  number  indicates 
how  many  times  the  question  was  selected.  For  each  question,  the  probability  for  that 
question  to  be  selected  “number  of  selections”  times  was  computed,  i.e.,  the  probability  that 
the  question  is  selected  the  number  of  times  it  was  in  fact  selected,  if  the  selections  were 
made  randomly;  that  number  was  called  the  “Random  Selection  probability”.  The 
complement  of  that  probability  was  then  computed  and  defined  as  the  “Core  Confidence 
metric”.  Thus  the  core  confidence  metric  represents  the  level  of  confidence  that  a  given 
question  was  not  selected  “number  of  selection”  times  at  random.  Each  of  the  questions 
contained  in  the  list  of  14  had  a  core  confidence  metric  higher  than  99%  (see  figures  9  and 
10). 

The  question  selections  were  done  independently  from  one  program  to  the  next,  i.e.,  by 
different  teams  of  people.  How  independent  the  question  selections  were  from  one  program 
to  another  can  be  argued.  We  recognize  that  some  teams  may  have  been  influcenced  by  the 
selections  made  by  their  collegues  on  previous  teams,  however,  we  feel  that  this  influence  is 
offset  by  tendencies  of  our  colleagues  to  want  to  differentiate  themselves  and  do  better  than 
the  previous  team. 

The  core  was  identified  in  terms  of  questions  instead  of  criteria  for  two  reasons.  The  first 
reason  is  that  the  number  of  questions  seems  to  be  a  better  metric  for  assessing  the  effort 
required  for  the  contractors  to  respond  to  SDCE  than  the  number  of  criteria,  since  the 
contractors  generate  one  response  per  question,  rather  than  one  response  per  criterion,  at  least 
normally.  Furthermore,  our  experience  showed  that  the  number  of  question  metric  tended  to 
be  scrutinized  by  the  Source  Selection  Evaluation  Board  (SSEB),  more  than  any  other 
number  associated  with  SDCE  tailoring.  The  other  reason  for  defining  the  core  in  terms  of 
questions  rather  than  criteria  is  that  the  questions  constitute  the  main  tool  for  soliciting 
information  from  the  contractors,  therefore  the  selection  of  the  questions  is  more  important 
than  that  of  the  criteria  for  obtaining  the  right  information  from  the  contractors.  This  is 
especially  true  when  site  visits  are  not  held,  in  which  case  the  responses  to  the  questions 
become  the  main  source  of  inputs  for  the  evaluation. 
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•Total  #ofSDCE 
Questions  =  717 

•Average 
questionnaire  size 
=  62 

•Total  number  of 
SDCEs  counted  =  9 

•Probability  that 
the  same  question 
was  chosen  4  times 
is  less  than  .008 

•Probability  that 
the  same  question 
was  chosen  9  times 
is  less  than  lO'^-Q 


Figure  9.  Core  Confidence  Associated  with  SDCE  Core  Questions 
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We  define  three  variables: 

N  =  the  total  number  of  SDCE  questions  (=700). 

M  =  the  number  of  programs  that  tailored  the  SDCE  question  set  (=9). 
n  =  the  average  number  of  SDCE  questions  selected  by  each  program  (=60). 

If  we  let  p  equal  the  probability  that  a  given  question  was  selected  at  random  on  one  of  the 
programs  using  SDCE,  then  p  can  be  computed  as  follows: 


The  total  number  of  subsets  of  size  n  out  of  N  questions  is 


The  total  number  of  such  subsets  that  include  the  given  question  is 


N-i 


Thus, 


A^-1' 


n-lj  _  n  _  60 

fm  "  iv  ~  m 


Now  we  can  compute  the  Random  Selection  probability,  i.e.,  the  probability  that  a  given 
question  was  selected  at  random  by  at  least  k  of  the  programs  using  SDCE.  Using  the 
binomial  formula, 

"  fM^ 

P,iN,M,n)  =  J^  . 

j=k  \  J  J 

Finally,  we  can  define  the  Core  Confidence  metric  as  the  confidence  that  a  given  SDCE 
question,  selected  by  at  least  k  of  the  programs  using  SDCE,  was  not  selected  that  many 
times  at  random.  This  is  just 


C,(N,M,n)  =  1  -  P,{N,M,n)  =  ^  .  p^(l  -  pf'^ 

y=oV  J  J 

Only  those  SDCE  questions  with  a  core  confidence  greater  than  99%  have  been  included  in 
the  core  set. 
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7.  SDCE  Outside  the  Source  Selection 


SDCE  can  be  thought  of  as  having  two  components:  the  model  component  and  the 
process  component.  The  model  represents  years  of  experience  in  software,  and  encodes  the 
software  risks  experienced  by  the  many  authors  who  contributed  to  the  material  in  the  SDCE 
model.  As  such,  it  can  be  used  for  any  risk  assessment  exercise,  whether  or  not  the  exercise 
is  conducted  within  the  strict  constraints  of  a  source  selection.  The  process,  on  the  other 
hand,  was  designed  to  be  consistent  with  the  source  selection  process,  as  a  tool  for  ensuring 
that  the  selected  offerors  have  adequate  software  capabilities.  Therefore,  using  SDCE  in  a 
non-source  selection  situation  requires  modifications  to  the  SDCE  process. 

There  have  been  several  attempts  at  performing  SDCE  outside  the  strict  source  selection 
framework.  These  attempts  have  been  driven  by  two  distinct  motivations  corresponding  to 
two  different  SDCE  scenarios.  The  first  motivation  is  the  tight  schedule  of  source  selections 
in  the  current  environment  of  acquisition  reform,  which  has  made  it  very  hard  to  conduct  site 
visits  within  the  source  selection  schedule.  In  this  scenario,  scenario  1,  some  of  the  SDCE 
activities  would  be  decoupled  from  the  source  selection  to  allow  more  time  for  the 
evaluations,  but  SDCE  results  would  still  be  used  in  a  source  selection.  The  second 
motivation  is  the  need  to  perform  software  risk  identification  after  contract  award:  SDCE 
seems  like  a  good  tool,  the  use  of  which  could  be  extended  beyond  source  selection.  In  this 
scenario,  scenario  2,  SDCE  results  would  not  be  used  for  a  source  selection,  but  simply  to 
identify  risks. 

One  attempt  was  made  at  an  early-start  scenario  1  SDCE,  i.e.,  in  a  competitive 
environment.  It  amounted  to  starting  the  evaluation  portion  of  SDCE  before  the  official 
proposal  receipt  but  after  the  RFP  had  been  sent  out.  This  was  possible  because  the  source 
selection  was  a  downselection  and  all  the  offerors  were  known  in  advance.  Starting  the 
SDCE  in  advance  of  proposal  receipts  gave  the  SDCE  team  additional  time  to  perform  their 
initial  evaluation.  Other  problems,  however,  were  experienced,  due  to  the  fact  that  the 
proposals  were  not  available  with  the  SDCE  volume.  A  portion  of  the  SDCE  evaluation  had 
to  be  redone  after  the  proposals  arrived.  In  the  end,  the  site  visits  still  needed  to  be  scheduled 
within  the  discussions  timeframe,  which  was  too  short,  so  site  visits  were  not  held  after  all.  It 
is  recommended  that  when  SDCE  is  performed  in  support  of  a  source  selection,  it  be 
performed  in  accordance  with  the  process  described  in  the  pamphlet,  augmented  with  the 
guidelines  provided  here,  and  that  scenario  1  not  be  repeated. 

Two  post-award  scenario  2  SDCEs  were  contemplated,  both  were  postponed  and  have 
not  yet  been  rescheduled.  There  are  two  difficulties  associated  with  performing  post-award 
SDCEs.  The  first  difficulty  is  that  the  contractors  have  no  incentive  to  perform  an  SDCE  if 
they  do  not  stand  to  gain  from  performing  one  (or  lose  from  not  performing  one).  The 
incentive  is  clear  in  source  selection,  however  after  contract  award,  unless  an  award  fee  is 
tied  to  SDCE  and  spelled  out  in  the  contract,  the  incentive  is  not  as  strong.  The  second 
difficulty  stems  from  the  fact  that  contractors’  efforts  for  preparing  for  an  SDCE  performed 
during  source  selection  are  reimbursed  with  Bid  and  Proposal  (B&P)  funds,  whereas  when 
SDCE  is  performed  after  contract  award  or  as  part  of  an  ongoing  contract,  the  cost  to  the 
contractors  is  paid  for  by  contract  money. 

If  a  methodology  is  needed  for  post-award  risk  identification  activities,  SDCE  questions 
and  criteria  can  be  used,  but  the  process  should  be  modified.  In  particular,  the  following  3 
items  need  to  be  carefully  planned: 

1.  Timing  of  the  Evaluation.  When  SDCE  is  performed  after  award,  there  are  more 

options  for  when  to  conduct  the  assessment.  It  is  recommended  that  the  evaluation  be 

schedulded  at  some  point  in  the  program  schedule  when  some  program  artifacts  have 
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been  generated  but  before  it  becomes  too  late  to  make  changes  to  the  software,  for 
example  right  before  the  completion  of  preliminary  design. 

2.  Program  Artifacts.  The  process  should  be  tailored  to  take  advantage  of  the  fact  that 
after  contract  award,  artifacts  exist  from  the  contract  at  hand,  so  the  evaluated  material 
should  not  be  limited  to  past  project  data  and  proposal  data.  In  particular,  the  contractors 
should  be  requested  to  submit  documentation  from  their  program  in  support  of  the  SDCE 
questions  selected. 

3.  Government  /  Contractor  Information  Exchange.  The  Source  Selection  regulates 
information  exchanges,  discussions  and  dialogs  between  the  government  and  the 
contractors  during  a  source  selection,  as  a  result,  they  are  highly  formalized  and 
minimized.  Information  exchanges  are  an  important  component  of  the  fact  finding  used 
by  SDCE  to  identify  risk.  The  following  guidelines  are  recommended  in  order  to  reduce 
the  formalism  associated  with  the  fact  finding,  reduce  the  effort  required  from  the 
contractors,  and  increase  information  content. 

•  The  tailored  SDCE  questions  should  be  sent  to  the  contractors  in  advance,  however, 
the  contractor  responses  can  be  provided  less  formally  than  as  written  responses  that 
are  mailed  in  advance  of  the  evaluation.  In  particular,  the  responses  can  be  provided 
at  the  same  time  as  the  site  visit  is  held,  in  the  form  of  a  presentation,  with  each 
question  being  addressed  on  a  separate  chart. 

•  The  results  of  the  evaluation  should  be  discussed  with  the  contractors  as  soon  as 
possible  in  a  cooperative  contractor  /  government  setting,  consistent  with  the 
Integrated  Product  Team  philosophy. 

•  Although  contractor  /  government  dialogs  should  be  encouraged,  they  should  be 
bounded  in  schedule  and  focused  on  the  evaluation  results,  as  opposed  to  the  selection 
of  questions.  Discussions  regarding  question  selection  can  go  on  forever; 
furthermore,  they  risk  delaying  or  cancelling  the  actual  evaluation. 
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8.  Suggested  Modifications  to  AFMC  Pamphlet 


The  following  is  an  enumeration  of  recommended  additions  and  modifications  to  the  AFMC 

SDCE  pamphlet.  The  changes  reflect  the  guideines  and  lessons  learned  discussed  in  the 

previous  chapters. 

•  Changes  to  the  SDCE  Process 

Only  one  change  is  recommended  to  the  SDCE  process,  namely  the  timing  of  the  SDCE 
team  selection.  The  sequencing  of  activities  as  described  in  the  pamphlet  indicates  that 
the  selection  of  SDCE  team  members  should  be  performed  before  the  tailoring  of  the 
model.  As  mentioned  in  Chapter  4,  the  SDCE  team  can  be  thought  of  in  terms  of  two 
subteams:  the  subteam  that  plans  the  evaluation  and  the  subteam  that  performs  the 
evaluation.  Although  the  two  subteams  should  have  some  overlap  in  personnel,  they  do 
not  have  to  comprise  the  same  people,  nor  do  they  have  to  be  determined  at  the  same 
time.  In  fact,  it  is  very  difficult  to  finalize  the  composition  of  the  evaluator  subteam  too 
much  in  advance  of  the  actual  start  of  the  evaluations.  As  the  RFP  release  date  slips 
(which  often  happens)  so  does  the  source  selection  schedule,  and  with  it  the  availability 
of  the  potential  evaluators.  It  is  recommended  that  the  selection  of  the  planning  subteam 
be  accomplished  before  the  tailoring  of  SDCE,  and  that  the  selection  of  the  evaluator 
subteam  be  started  after  the  tailoring  of  the  model  and  completed  soon  after  RFP  release. 
This  would  allow  the  SDCE  evaluators  to  know  the  schedule  they  are  committing  to 
before  finalizing  their  commitment. 

•  Additions  to  SDCE  Model 

An  updated  version  of  the  SDCE  model  has  been  used  on  several  SMC  acquisitions.  The 
new  model  has  additions  to  FA6  reflecting  new  software  technology  trends,  and  additions 
in  the  areas  of  reuse  and  software  architecture  to  support  the  increasing  emphasis  on  open 
system  architectures  and  reuse  [ref  7]. 

•  Improved  RFP  Templates 

A  new  template  was  developed  for  the  offerors  to  list  all  substantiating  documents 
submitted,  and  cross-reference  them  to  the  applicable  SDCE  sections.  The  Capability 
Definition  Matrix  was  modified  to  delete  references  to  specific  military  specifications 
and  standards.  Finally  the  Capability  Implementation  Matrix  needs  to  be  modified  to 
disallow  the  use  of  “I”  for  capability  Implemented  but  no  sample  provided,  and  to  delete 
references  to  specific  military  standards.  Appendix  A  “Example  SDCE  Instructions  to 
Offerors”  contains  updated  versions  of  the  SDCE  RFP  templates. 

•  Improved  Evaluation  Templates 

Evaluation  templates  were  provided  in  volume  2  of  the  SDCE  pamphlet  as  an  example  of 
templates  that  could  be  helpful.  It  is  recommended  that  the  format  of  the  evaluation 
templates  contained  in  the  pamphlet  be  aligned  with  the  format  of  the  source  seleciton 
results  as  described  in  chapter  5.  It  is  also  recommended  that  the  extensive  roll-up 
scheme  they  imply  be  abandoned,  and  that  SDCE  results  be  developed  at  the  CCA  level, 
then  rolled  up  to  the  overall  SDCE  level,  as  discussed  in  in  chapter  4  in  “Levels  at  which 
SDCE  results  are  developed.”  Appendix  B  “SDCE  Evaluation  Templates”  contains 
updated  versions  of  the  SDCE  evaluation  templates. 
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Appendix  A:  Sample  SDCE  Instructions  to  Offerors 
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Software  Development  Capability  Evaluation  (SDCE)  RFP  Instructions 


The  following  information  in  direct  support  of  the  SDCE  shall  be  submitted  with  the 
proposal. 

1.  Questionnaire  Responses 

In  Volume  VI,  the  Offeror  shall  provide  responses  to  the  questions  that  are  included  in  this 
attachment.  To  reduce  the  SDCE  preparation  effort  and  duplication,  responses  to  the 
questions  may  be  provided  directly  in  the  documentation  accompanying  the  proposal,  such  as 
the  Integrated  Master  Plan  (IMP),  Integrated  Master  Schedule  (IMS),  or  other  proposal 
volumes.  When  responses  to  the  SDCE  questions  are  provided  in  other  proposal 
information,  specific  page  number  and  paragraph  references  shall  be  provided  with  the 
response  to  the  question.  Information  shall  be  provided  for  the  Offeror,  any  team  member,  or 
subcontractor  who  will  have  a  role  in  the  software  development  effort.  Common  processes 
require  only  one  response.  NOTE:  the  term  “process”  in  the  SDCE  context  includes 
procedures,  standards,  methodologies,  tools,  etc. 


2.  Substantiating  Documents 

Substantiating  documents  shall  be  submitted  for  all  planned  processes,  whether  employed  by 
the  prime  Offeror,  subcontractors,  or  other  team  members.  Substantiating  documents  are 
intended  to  demonstrate  institutionalization  of  proposed  processes.  These  documents  shall 
include: 

a.  Copies  of  corporate  software-related  process  descriptions  that  are  to  be  used  in  the 
XYZ  program. 

b.  Copies  of  documents  that  provide  evidence  of  current  or  prior  use  of  the  proposed 
processes.  Specific  page  number  and  paragraph  references  for  each  substantiating 
document  shall  be  provided  with  the  response  to  each  question. 


3.  SDCE  Forms 

The  following  forms  shall  be  completed  and  submitted  with  the  proposal: 

a.  List  of  substantiating  documents. 

b.  Cover  Sheet  for  each  substantiating  document  submitted. 

c.  Capability  Definition  Matrix  (one  per  Critical  Capability  Area), 

d.  Capability  Implementation  Matrix  (one  per  Critical  Capability  Area),  and 

4.  SDCE  Detailed  Responses 

The  Offeror  shall  supply  information  as  described  below: 


42 


4.1  Overview  of  the  Proposed  Software  Development  Effort 

The  Offeror  shall  provide  an  overview,  not  to  exceed  four  (4)  pages,  that  addresses  the  total 
software  development  effort,  the  organization  of  the  Offeror  team,  and  the  task  and 
responsibility  distribution  among  the  team  members  for  the  entire  life  cycle  of  the  project; 
and  the  processes  used  to  manage  and  control  team  member  performance.  The  purpose  of 
this  overview  is  to  provide  a  foundation  for  review  of  the  SDCE  questionnaire  responses.^  All 
information  in  this  overview  shall  be  consistent  with  information  provided  in  the  Offeror’s 
planning  documentation  for  the  program,  and  shall  reference  such  information  where 
appropriate.  The  Overview  is  included  in  the  overall  page  limit. 

4.2  SDCE  Questionnaire  Responses 
4.2.1  General  Information 

Responses  to  the  questions  shall  be  concise  and  unambiguous.  Common  processes  require 
only  one  response.  If  processes  are  partially  but  not  fully  common,  or  if  different  processes 
are  used  by  the  Offeror  and  any  team  members  or  subcontractors,  these  differences  shall  be 
identified  and  each  process  described. 


4.2.2  Response  Page  Limits  and  Organization 

The  following  instructions  shall  be  applied  in  preparing  responses: 

a.  The  total  number  of  pages  permitted  for  the  SDCE  reponses  to  all  CCs  combined  shall 
not  exceed  the  number  of  SD&  questions  (give  a  specific  number).  Note  that  the  number  of 
pages  allocated  for  any  given  response  is  left  up  to  the  Offeror. 

b.  Substantiating  information  referenced  in  the  responses  is  not  subject  to  the  page  limit. 
Specific  volume,  page,  and  paragraph  numbers  are  required  to  be  supplied  in  the  reference. 
References  shall  be  made  only  to  documents  delivered  as  part  of  the  proposal. 

c.  Past  project  samples  shall  be  limited  to  (1  or)  2  samples  per  CC. 

4.2.3  Substantiating  Documents 
4.2.3. 1  Types  of  Documents 

Substantiating  documents  are  included  in  Vol  VII  and  are  intended  to  demonstrate  experience 
with  and  commitment  to  the  proposed  processes.  Substantiating  documents  shall  be 
submitted  for  each  critical  capability  (CC)  response.  These  documents  shall  cover  all 
planned  processes,  whether  employed  by  the  prime  contractor,  subcontractors,  or  other  team 
members.  Where  common  processes  are  planned,  evidence  of  use  by  all  relevant  parties  (i.e., 
prime  contractor,  subcontractors,  and  other  team  members)  shall  be  supplied.  Samples  of 
existing  processes  shall  be  relevant  to  the  XYZ  program’s  needs  and  relate  to  an  SDCE 
question  (see  Figures  3a  and  3b,  Capability  Definition  Matrix).  These  documents  shall 
include: 

a.  Process  Descriptions  and  Plans  for  Use — Copies  of  corporate  software-related  process 
descriptions  that  are  relevant  to  XYZ  PROGRAM.  If  different  processes  are  to  be  employed 
by  subcontractors  and/or  team  members,  these  shall  be  supplied  as  well. 
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b.  Project  Samples — Copies  of  documents  that  provide  evidence  of  actual  use  of  the 
proposed  processes  (e.g.,  current  or  historical  development  schedules,  software  development 
plans,  software  requirements  specifications,  test  and  integration  plans,  procedures,  etc.).  Past 
project  samples  shall  be  limited  to  (1  or)  2  samples  per  CC. 

c.  New  Process  Rationale — ^For  new  processes  not  yet  implemented,  describe  the 
benefits  and  risks  of  using  the  new  processes  and  the  rationale  for  employing  them  in  lieu  of 
providing  examples  of  past  application. 

See  Figure  1  for  examples  of  substantiating  documents. 

5.  Capability  Cross-Reference  Forms  and  Sample  Data  Cover  Sheet 

The  forms  listed  below  shall  be  completed  and  submitted  with  the  SDCE  responses  and 
substantiating  information.  Figures  2a,  3a,  4a  and  5a  contain  blank  copies  of  these  forms 
and  figures  2b,  3b,  4b  and  5b  contain  examples  of  each. 

a.  Cover  Sheets  for  substantiating  documents  (one  for  each  document  submitted);  see 
Figures  2a  and  2b. 

b.  Capability  Definition  Matrix  (one  per  Critical  Capability  Area  (CCA));  see  Figures  3a 
and  3b. 

c.  Capability  Implementation  Matrix  (one  per  CCA);  see  Figures  4a  and  4b. 

d.  A  list  of  all  Substantiating  Documents  must  be  provided.  The  list  must  contain 
document  title,  its  unique  ID  and  the  Critical  Capability  (CC)  that  it  addresses.  The  list  of 
Substantiating  Documents  shall  be  organized  either  by  ID  number  or  by  CC  (see  Figures  5a 
and  5b  below). 
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PROJECT  SAMPLE  DOCUMENT 

CCA/CC 

Software  development  plans 

3.1, 5.7 

Program  organizational  charts 

1.1.1,  1.1.2 

Organizational  charts  from  program  manager  through  working-level  software  engineers 

1.1.1,  1.1,2 

Contract  work  breakdown  structures  covering  software  development 

1.2.2,  3.1.2 

Software  work  packages 

1.2.3 

Cost/schedule  control  system  criteria  reports  applied  to  software 

1.2,3,  1.2.4 

Cost  performance  reports  applied  to  software 

1.2.3 

Svstems  through  detailed  software  development  schedules 

1.2.3,  1.2.4 

Svstems  engineering  master  schedules  showing  software  events  and  completion  criteria 

1.2.4 

Subcontractor  RFPs  and  SOWs  defining  software  tasks 

1.3 

Maintenance  contracts  for  providing  software  rights  for  post  deployment 

1.4 

Risk  management  plans  covering  software 

1.5,  3.2 

Systems  and  subsystem  specifications 

2.1 

Tradeoff  study  reports  addressing  software 

2.1.5,  1.5.2 

Requirements  traceability  matrices/tables 

2.1.4,  2.1.5 

Design  review  minutes 

2.2, 2.3 

Interface  control  specifications/documents 

2.1.1 

Svstems  engineering  master  plans 

2.5.3 

Systems  engineering  staffing  plans/final  reports 

2.5.4 

Test  and  integration  plans 

2.6.1,  2.6.2 

Reuse  plans  covering  software 

2.7 

Reuse  trade-off  reports 

2.7 

Software  size,  effort,  schedule,  and  cost  estimates 

3.1.1 

Past  actual  productivity  rates 

3.1.1 

Software  tree  structures  (CSCls  through  units) 

4.7 

Software  status  reports 

3.2.2 

Software  requirements  specifications 

3.3.1 

Software  development  folders/f i  les 

3.4.1,  4.7 

Peer  review  minutes/reports 

3.5.1, 4.5.2 

Software  integration  and  test  plans.  Software  Test  Procedures 

3.6 

Software  quality  assurance  plans 

4.1.1 

Software  discrepancy  reports 

4.1.3 

Defect  prevention  plans 

4.3.1 

Software  development  metrics 

4.4 

Peer  review  plans 

4.5.1 

Internal  independent  verification  and  validation  plans 

4.6.1 

Software  configuration  management  plans 

4.7.1 

Software  development,  integration,  and  test  facilities  plans 

5.2 

Software  training  plans 

5.3.1 

Software  staffing  plans,  including  actual  staffing  profiles  on  completed  programs 

5.4.1 

Software  process  improvement  plans 

5.6.1 

Integrated  Master  Plans,  Integrated  Master  Schedules 

1.2,  3.1.  3.2 

Internal  standards  and  procedures  documents  (for  software  development,  quality  assurance, 
configuration  management,  systems  engineering,  etc.) 

3.5, 4. 1,4.7,  4.8 

Figure  A- la.  Examples  of  Substantiating  Documents 


Document  ID 

Document  Title 

Applicable  CC(s) 

LM, 

F-22  Schedule  Volatility  Metrics 

4.4.1 

LM, 

F-22  SDP 

3.1.4 

LM, 

GPS  SDP 

3.1.4 

• 

• 

• 

• 

• 

• 

• 

• 

• 

etc. 

etc. 

etc. 

Figure  A-2b.  Example  List  of  Substantiating  Documents 
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Cover  Sheet  for  Substantiating  Document 


Sample  ID: 


Sample  Project  Name  and  unique  ID: 


Sample  Project  Customer: 


Critical  Capability (ies): 


Explain  why  your  experience  on  the  sample  project  is  relevant  to  the  proposed  project. 


ATTRIBUTES 


Application  Domain 


Product  Type 


Acquisition  Phase^ 


Software  Development  Phase(s) 


Award  Date 


Contract  Duration 


Current  Project  Phase/ 
Contract  Month^ 


Prime/Subcontractors^ 


Software  KSLOC"^ 


Language(s)  and  Percentages 


Target  Processor(s)/OS(s) 


Applicable  Standards 


PROPOSED  PROJECT 


SAMPLE  PROJECT 


^For  “Proposed  Project,”  phase(s)  in  which  Critical  Capability(ies)  are  to  be  used;  for  “Sample  Project,”  phase 
in  which  sample  was  generated. 

^Phase/month  of  the  Sample  Project  as  of  the  current  date. 

^Contractors  developing  the  software  products  specified  in  the  “Product  Type”  row 
^otal  number  of  KSLOC  for  software  specified  in  the  “Product  Type”  row 


Figure  A-3a.  Substantiating  Document  Cover  Sheet 


Cover  Sheet  for  Substantiating  Document 


Contractor:  Team  A 

Sample  ID:  IDX^^c 

Sample  Project  Name  and  unique  ID:  Project  X  Design  Complexity  Metrics 

Sample  Project  Customer:  U.S.  Air  Force  Space  and  Missile  Systems  Center 

Critical  Capability(ies):  4.4.2  Metrics  Application 

Title  of  Sample:  Project  X  Software  Development  Metrics  Reports 

Explain  why  your  experience  on  the  sample  project  is  relevant  to  the  proposed  project. 

Object-oriented  methods  and  metrics  were  used  on  the  sample  project.  The  same  object-oriented  methods  and  metrics  are 
planned  for  use  on  the  proposed  project. 

ATTRIBUTES 

PROPOSED  PROJECT 

SAMPLE  PROJECT 

Application  Domain 

Weather  Satellite 

Communications  Satellite 

Product  Type 

Ground  System  (Command  and  Control) 

Ground  System  (Command  and  Control) 

Acquisition  Phase  ^ 

HMD 

EMD 

Software  Development  Phase(s) 

Design;  Coding  and  Unit  Test 

Coding  and  Unit  Test,  Increments  1  and  2 

Award  Date 

1/94 

Contract  Duration 

8  Years 

5  Years 

Current  Project  Phase/ 

Contract  Month^ 

• 

EMD:  Between  System  PDR  and  System 
CDR/Month  24 

Prime/Subcontractors^ 

2  Software  Subs 

Prime  &  1  Software  Sub 

Software  KSLOC'* 

750 

500 

Language(s)  and  Percentages 

Ada  95:  90% 

C++:  10% 

FORTRAN  77:  75%  C++:  25  % 

Target  Processor(s)/OS(s) 

RISC  6000/UNIX 

VAX  6200/VMS  6.2 

Applicable  Standards 

IEEE  1498 

DOD-STD-2167A  &  2168 

^For  “Proposed  Project,”  phase(s)  in  which  Critical  Capability(ies)  are  to  be  used;  for  “Sample  Project,”  phase 
in  which  sample  was  generated. 

^Phase/month  of  the  Sample  Project  as  of  the  current  date. 

^Contractors  developing  the  software  products  specified  in  the  “Product  Type”  row 
^otal  number  of  KSLOC  for  software  specified  in  the  “Product  Type”  row 


Figure  A-3b.  Example  Substantiating  Document  Cover  Sheet 
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CAPABILITY 

DEFINITION  REFERENCE  DOCUMENTS  LOCATION  OF  CAPABILITY  DESCRIPTION  WITHIN 

MATRIX  REFERENCE  DOCUMENTS 


CAPABILITY  - - 

DEFINITION  REFERENCE  DOCUMENTS  LOCATION  OF  CAPABILITY  DESCRIPTION  WITHIN 

MATRIX  REFERENCE  DOCUMENTS 


CAPABILITY 

IMPLEMENTATION  PROJECTS  IMPLEMENTED  ON 


CAPABILITY 

IMPLEMENTATION  PROJECTS  IMPLEMENTED  ON 


rsi 


Figure  A-5b.  Example  Capability  Implementation  Matrix 


Appendix  B:  SDCE  Evaluation  Templates 
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Figure  B-1 .  Revised  CCA  Score  Sheet 


SDCE  SCORE  SHEET 


Figure  B-2.  Revised  SDCE  Score  Sheet 


Appendix  C:  List  of  Past  SDCEs 


The  table  below  lists  the  past  12  SDCEs  that  were  tracked  and  used  to  derive  the  guidelines 
contained  in  this  document.  For  each  SDCE  application,  the  table  lists  the  contract  name,  the 
associated  program,  and  the  general  time  frame  during  which  SDCE  was  performed.  All  but 
one  of  the  applications  below  were  associated  with  source  selections.  In  one  case,  SDCE 
was  used  to  evaluate  a  contractor  that  was  being  considered  for  a  sole  source  contract.  Data 
from  that  application  was  used  only  for  developing  recommendations  relating  to  model 
tailoring. 


Contract  Name 

Proeram  Name 

SDCE  Evaluation  Date 

Small  Tactical  Terminal 
(STT) 

Defense  Meteorological 

Satellite  Program  (DMSP) 

FY94 

Advanced  Communication 
Management  System 
(ACMS) 

Milstar 

FY94 

Cloud  Depiction  and 
Forecasting  System  (CDFS) 
II 

Defense  Meteorological 

Satellite  Program  (DMSP) 

FY94 

Operation  Control  System 
(OCS)  support 

Global  Positioning  Satellites 
(GPS) 

FY95 

Range  Standardization  and 
Automation  (RSA)  Phase  II 

Air  Force  Satellite  Control 
Network  (AFSCN) 

FY95 

Network  Operation 

Upgrade  Contract  (NOUC) 

Air  Force  Satellite  Control 
Network  (AFSCN) 

FY95 

Range  Communication  Data 
Contract  (RCDC) 

Air  Force  Satellite  Control 
Network  (AFSCN) 

FY95 

AFSCN  Command  and 
Control  Sustainment 

Contract  (CCSC) 

Air  Force  Satellite  Control 
Network  (AFSCN) 

FY95 

SBIRS  High  downselect 

Space  Based  Infra  Red  Sensors 
(SBIRS) 

FY96 

GPS  Block  IIF 

Global  Positioning  Satellites 
(GPS) 

FY96 

Classified  Contract 

Classified  NRO  program 

FY96 

Low  Cost  Contract  Vehicle 
(LCCV)  EELV  downselect 

Expendable  Evolving  Launch 
Vehicle  (EELV) 

FY97 

Global  Broadcasting 

Services  (GBS) 

Milstar 

FY97 
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Appendix  D:  List  Of  Acronyms 


AFFARS 

Air  Force  Federal  Acquisition  Regulation  Supplement 

AFMC 

Air  Force  Materiel  Command 

ASC 

Aeronautical  Systems  Center 

ASP 

Acquisition  Strategy  Panel 

CC 

Critical  Capability 

CCA 

Critical  Capability  Area 

CMM 

Capability  Maturity  Model 

CR 

Clarification  Request 

DR 

Deficiency  Report 

ESC 

Electronic  Systems  Center 

FA 

Functional  Area 

FAR 

Federal  Acquisition  Regulation 

OPR 

Office  of  Prime  Responsibility 

RFP 

Request  For  Proposals 

SDCCR 

Software  Development  Capability  Capacity  Review 

SDCE 

Software  Development  Capability  Evaluation 

SEI 

Software  Engineering  Institute 

SMC 

Space  and  Missile  Systems  Center 

soo 

Statement  Of  Objectives 

ss 

Source  Selection 

SSEB 

Source  Selection  Evaluation  Board 

TRD 

Technical  Requirements  Document 
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